Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Environmental influences on ophthalmic conditions: A scoping review.

BACKGROUND: Environmental factors have been implicated in various eye pathologies. The purpose of this review is to synthesise the published research on environmental effects on eye disease. METHODS: Four databases were searched for terms relating to environmental exposures and ophthalmic disease. Titles and abstracts were screened followed by full-text review. Data was extracted from 118 included studies. Quality assessment was conducted for each study. RESULTS: Air pollutants, including nitrogen dioxide, nitrites, sulphur dioxide, particulate matter, carbon monoxide, ozone and hydrocarbons are associated with ocular conditions ranging from corneal damage to various retinopathies, including central retinal artery occlusion. Certain chemicals and metals, such as cadmium, are associated with increased risk of age-related macular degeneration. Climate factors, such as sun exposure, have been associated with the development of cataracts. Living in rural areas was associated with various age-related eye diseases whereas people living in urban settings had higher risk for dry eye disease and uveitis. CONCLUSION: Environmental exposures in every domain are associated with various ophthalmic conditions. These findings underscore the importance of continued research on the interplay between the environment and eye health.

Home Monitoring of Glaucoma Using a Home Tonometer and a Novel Virtual Reality Visual Field Device: Acceptability and Feasibility.

OBJECTIVE: Our aim was to assess the acceptability and feasibility of iCare HOME tonometer (HT) and Virtual Field (VF) devices in the home monitoring of glaucoma. DESIGN: Prospective feasibility and acceptability study. SUBJECTS: Twenty patients (39 eyes) with primary open-angle glaucoma, open-angle glaucoma, ocular hypertension, or suspected glaucoma. METHODS: Patients were trained and instructed to bring 2 devices home for 1 week and use the HT 4 times/day for 4 days and the VF 3 times total. MAIN OUTCOME MEASURES: For acceptability, we conducted satisfaction surveys and semistructured, qualitative interviews with a thematic analysis. Feasibility was assessed by device usage and quality of tests. RESULTS: Most patients (73.7%) felt that the HT was easy to use, and 100% of them found the HT useful. All patients (100%) felt that VF was easy to use, and 94.4% of them found the VF useful. All patients (100%) obtained acceptable intraocular pressure and completed a VF test at home. We identified 4 key themes, with 33 subthemes. The key themes include the following: (1) advantages of home monitoring; (2) difficulties with home monitoring; (3) future considerations in home monitoring; and (4) the experience of patients with glaucoma. CONCLUSIONS: The HT and VF were acceptable and feasible in a small cohort of motivated subjects. Patients were able to perform these tests proficiently at home, and they were generally enthused to obtain more data about their intraocular health, as it allowed them a heightened sense of security and insight about their chronic disease, as well as a reduction in foreseeable barriers to care. Home monitoring may also improve upon glaucoma care by enhancing patient empowerment and fostering community bonds. The VF should be further evaluated to ensure validity. FINANCIAL DISCLOSURE(S): Proprietary or commercial disclosure may be found after the references.

Diversification of Retinoblastoma Protein Function Associated with Cis and Trans Adaptations.

Retinoblastoma proteins are eukaryotic transcriptional corepressors that play central roles in cell cycle control, among other functions. Although most metazoan genomes encode a single retinoblastoma protein, gene duplications have occurred at least twice: in the vertebrate lineage, leading to Rb, p107, and p130, and in Drosophila, an ancestral Rbf1 gene and a derived Rbf2 gene. Structurally, Rbf1 resembles p107 and p130, and mutation of the gene is lethal. Rbf2 is more divergent and mutation does not lead to lethality. However, the retention of Rbf2 >60 My in Drosophila points to essential functions, which prior cell-based assays have been unable to elucidate. Here, using genomic approaches, we provide new insights on the function of Rbf2. Strikingly, we show that Rbf2 regulates a set of cell growth-related genes and can antagonize Rbf1 on specific genes. These unique properties have important implications for the fly; Rbf2 mutants show reduced egg laying, and lifespan is reduced in females and males. Structural alterations in conserved regions of Rbf2 gene suggest that it was sub- or neofunctionalized to develop specific regulatory specificity and activity. We define cis-regulatory features of Rbf2 target genes that allow preferential repression by this protein, indicating that it is not a weaker version of Rbf1 as previously thought. The specialization of retinoblastoma function in Drosophila may reflect a parallel evolution found in vertebrates, and raises the possibility that cell growth control is equally important to cell cycle function for this conserved family of transcriptional corepressors.